test: convert silent t.Skip to t.Fatal in baseline-scan checks#8
Closed
omercnet wants to merge 1 commit into
Closed
test: convert silent t.Skip to t.Fatal in baseline-scan checks#8omercnet wants to merge 1 commit into
omercnet wants to merge 1 commit into
Conversation
Three sites in test/e2e/golang and one in test/e2e/frontend were silently skipping tests when their baseline assertion failed. This masks real regressions: - test/e2e/golang/golang_test.go (TestGoBaselineImages, TestMultiBinaryImages, TestDistrolessImages): each test selects a baseline image specifically because it carries known CVEs. If the baseline scan returns zero vulnerabilities, the test should FAIL loudly — it indicates a scanner regression, a stale Trivy DB, an upstream image rebuild that resolved CVEs, or a network failure during DB download. Silently skipping made the test green when in fact the test was unable to assert anything. - test/e2e/frontend/frontend_test.go (TestMultiplatformFrontend): when *every* platform's report generation fails the test was skipping with a hand-wavy 'possible disk space or network issue' message. Per-platform failures are already surfaced via t.Logf within the loop; if the cumulative result is zero successful reports the failure is real, not flake. Each Fatalf message includes enough context (image ref, mode) for the on-call to triage without re-reading the test. Also adds a reason string to the legitimate skip in integration/singlearch/patch_test.go where local-name images are not testable on non-docker buildkit drivers — this skip is correct, but previously had no message which made gotestsum output unreadable. No production code changes.
Author
|
Superseded by upstream PR project-copacetic#1578 — closing this duplicate. CI never ran on this fork (no oracle-vm runners registered). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
Three sites in `test/e2e/golang` and one in `test/e2e/frontend` were silently skipping when their baseline assertion failed. This masks real regressions:
`test/e2e/golang/golang_test.go` (`TestGoBaselineImages`, `TestMultiBinaryImages`, `TestDistrolessImages`)
Each test selects a baseline image specifically because it carries known CVEs. If the baseline scan returns zero vulnerabilities, the test should FAIL loudly — it indicates one of:
Silently skipping made the test green when in fact the test was unable to assert anything.
`test/e2e/frontend/frontend_test.go` (`TestMultiplatformFrontend`)
When every platform's report generation fails, the test was skipping with a hand-wavy "possible disk space or network issue" message. Per-platform failures are already surfaced via `t.Logf` within the loop; if the cumulative result is zero successful reports the failure is real, not flake.
`integration/singlearch/patch_test.go` (legitimate skip, just needs a message)
The skip for local-name images on non-docker buildkit is correct — local-only images can only be tested with docker:// daemon. But the previous `t.Skip()` had no message, making gotestsum output unreadable. Adds a `Skipf` with image ref + current `COPA_BUILDKIT_ADDR`.
Risk
Could turn currently-green CI red if the underlying assumption is wrong (e.g., a baseline image was rebuilt upstream and now has 0 CVEs legitimately). If that happens, the right response is to update the baseline test data, not to re-silence the failure.
Test plan
Summary by cubic
Convert silent
t.Skiptot.Fatalin baseline-scan checks so real regressions fail fast in CI. Also add a message to one legitimate skip for clearer logs. No production code changes.test/e2e/golang/golang_test.go: Fail witht.Fatalfwhen a baseline scan returns 0 vulns; includes image context.test/e2e/frontend/frontend_test.go: Fail when zero reports are generated across all platforms.integration/singlearch/patch_test.go: Addt.Skipfmessage for local-only images on non-docker://buildkit.Written for commit 66b989e. Summary will update on new commits. Review in cubic